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The present invention relates to a method of selecting among N "Spatial Video 
CODECS" where N is an integer number greater than 1, the optimum "Spatial 
Video CODEC" for a same input signal I. In this new technique (hereafter referred 
to as "Dynamic Coding") for digital video coding, "Spatial Video CODEC" is 
understood as the combination of any transform of the input signal, followed by a 
quantization of the transform coefficients and a corresponding entropic coder. 

Video Coding is an important issue in all application fields where digital video 
information has to be stored on a digital support or transmitted over digital net- 
works. Several solutions have been proposed in the last 20 years and standardiza- 
tions efforts have been undertaken to define a unified syntax. 



Standard video coding schemes have a rigid structure. They take into account the 
context of specific, well-defined applications requiring video coding, and propose 
an optimized, albeit limited, solution. This explains the number of existing inter- 
national recommendations that have been defined for specific applications. For 
20 example, the ITU-T H.261 standard is designed for tele-conferencing and video- 
telephony applications, MPEG-1 for storage on CD-ROM, MPEG-2 for wide- 
band TV broadcast, MPEG-4 for low-bitrate coding with multimedia fiuictional- 
ities and H264 for very low bit-rate video coding. 

25 The strategy adopted by classical video coding schemes is prompted by the fact 
that no a single universal coding technique can be applied with optimum results in 
every context. In fact, the performance of a "Spatial Video CODEC" depends on 
several application specific parameters, such as: the type of the data to be com- 
pressed (still pictures, video, stereo imagery, and so on), the nature of the visual 
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data (natural, synthetic, text, medical, graphics, hybrid), the target bitrate, the 
maximum acceptable delay (ranging from few milliseconds to off-line), the mini- 
mum acceptable quality (spatial and temporal resolution), the type of communica- 
tion (point to point, broadcast, multi-point to point, etc.), and the set of fimction- 
alities required (scalability, progressiveness, etc.). Often these parameters such as 
the nature of the input signal or the available bandwidth may change in time with 
a consequent variation of the perforaiances of the selected "Spatial Video 
CODEC". In the following table, major specifications for a few of the most critical 
applications for video coding are listed. Reads ♦'Mbps" Mega bit per second, 
"Kbps" kilobit per second, "Sps" frame per second, "MP2MP" multi point to 
multi point, "P2P" point to point, "P2MP" point to multipoint 





Type of data 


Nature 
of data 


Target 
bitrate 


Max 
delay 


Minimum 
quality 


Type of 
comm. 


Functionalities 


Video Surveil- 
lance 


Low_motion 
Video 


Natural 


3-6 
Mbps 


300 
ms 


25-50 
fps, 
FULL 


MP2M 
P 


Fast frame 
access 


Video teleph- 
ony 


Static Video 


Natural 


<500 
Kbps 


200 

ms 


10 fps, 
QCIF 


P2P 


Scalability 


Telemedicine 


Video 
Still pictures 


Hybrid 
Text 


>5 
Mbps 


< 1 
sec 


50 fps, 
FULL 


MP2M 
P 


Scalability, 
Editing,... 


Digital TV 


High_motion 
Video 


Natural, 

text, 
synthetic 


< 1 
Mbps 


< 1 
sec 


25 fps, 
FULL 


P2MP 


Record, fast 
frame access 


Corporate TV 


Natural 
Text Syn- 
thetic 


Natural^ 
text, 

synthetic 


2 

Mbps 


< I 
sec 


50 fps, 
FULL 


P2MP 


Scalability 


Video confer- 
ence 


Natural 
Text 


Natural, 

text, 
synthetic 


512 
Kbps 


< 1 
sec 


25 fps, 
QCIF- 
CIF 


MP2M 
P 


Scalability 



Given the wide variations in the requirements from application to application, it is 
15 clear that a coding scheme nmed for a specific application will be superior, for 
that application, to an universal coding technique that attempts to find a suitable 
compromise among different consh^ints. However, even the optimum "Spatial 
Video Video CODEC" for a specific set of consti^nts may be a sub-optimal solu- 
tion when the parameters of the application are allowed to change tiirough time. 
20 For example, in several multimedia scenarios, the video input combines static 
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scenes to highly dynamic ones. Moreover, the sequences may be natural images, 
or synthetic scenes or combination of the two (text, graphs, and natural images) 
with a consequent variation of the statistical properties of the input signal. 

5 The present invention proposes a method which is a suitable solution to achieve 
an optimum video coding despite the changes of the above discussed properties of 
the input signal. 

The method is defined by the characterizing portion of claim 1, while the depend- 
10 ent claims propose various embodiments. 

The new proposed paradigm to Video coding is based on the following idea: dy- 
namically evaluate the performances of several coders given the input data and the 
external constraints of the application. This evaluation is performed on a block- 
15 based basis. The input image can be organized as a set of rectangular blocks 
whose size can range from 2 by 2 pixels to the size of flie input image. The results 
of such an evaluation are used to select the best performing among the available 
"Spatial Video CODECs" for that given input data. A prototype implementing this 
strategy has proved its superiority against standard approaches. 

20 

The invention will now be disclosed with the help of the accompanying drawings. 

Figure 1 shows the general structure of any video coding scheme, 

25 Figure 2 shows MPEG-like simplified video coding scheme, 

Figure 3 shows how the proposed "Dynamic Coding" fits in a standard video cod- 
ing scheme. 
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Figure 4 shows the scheme of the Dynamic Coding base block. 

Figure 5 shows examples of four different Normalized "Spatial Video CODECs", 

Figure 6 shows a statistical evaluation of the "Spatial Video CODEC" alignment 
error, 

Figure 7 shows a standard procedure to compute the Rate and the Distortion of the 
"Spatial Video CODEC", 

Figure 8 shows a proposed Rate Distortion prediction scheme. 

Figure 9 shows a performance comparison of three "Spatial Video CODECs" on a 
frame basis. 

Figure 10 shows the frequency of the selection of a "Spatial Video CODEC" dur- 
ing the encoding of a 10 minutes typical TV sequence. 

In order to efficiently encode any digital signal it is important to exploit all the 
redundancies that the signal contains. When the input signal is a video signal there 
are three different kind of redundancies that can be exploited: the spatial, the tem- 
poral and the statistical redundancy. All existing video coding schemes apply dif- 
ferent techniques to exploit more or less all these redundancies. Based on these 
observations the general scheme of any video coding is represented in Figure 1. 
Each of the three main modules can be performed differently. As already dis- 
cussed, the overall performances of any "Spatial Video CODEC" depends on the 
nature of the input signal, on the constraints of the application and clearly on the 
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combination of the right tools for each of the three modules represented in Figure 
1. 

Because of practical constraints, it is not possible to imagine that all the possible 
5 solutions can be evaluated. Thus the approach adopted by the standards is to fix 
one combination that provides the best compromise in some specific scenario. 
This constraints the efficiency of the "Spatial Video CODEC", but simplifies its 
implementation. What is lacking in order to dynamically adapt the scheme to the 
external constraints is an efBcient prediction of the performances of each tool. 

10 

We will explain how it is possible to evaluate several tools to dynamically select 
the one that exploits optimally the spatial redundancy of the signal, by taking into 
account the properties of the previous tool used to exploit the temporal redun- 
dancy and the following tool used to exploit the statistic redundancy. This evalua- 
15 tion is fast and efficient and boosts the coding perforaiances of standard ap- 
proaches. 

Hereafter, we refer to both the spatial and entropic redundancy modules as "Spa- 
tial Video CODEC". In this document, as previously mentioned, "Spatial Video 
20 CODEC" is understood as the combination of any transform of the input signal, 
followed by a quantization of the transform coefficients and a corresponding en- 
tropic coder. 

A simplified version of MPEG-like standard video encoder is shown in Figure 2, 
25 where the Temporal redundancy is exploited with a block-matching approach 
(motion estimation and motion compensation), the spatial redundancy is exploited 
by DCT. Standard video coding schemes use the block based DCT transform in 
order to exploit the spatial redvmdancy present in the signal. The input signal is a 
natural image or a difference of two natural images. The input is divided into 
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15 



square blocks and each block is DCT transformed. The resulted coefficients are 
quantized to increase their redundancy. These quantized coefficients are encoded 
with a statistical coder and the results represent the output bitstream. By de- 
quantizing (scaling back) the quantized coefficients and by applying the inverse 
5 DCT transform, the decoded frame is obtained. This is stored and it is used as 
reference to predict the following frames of the input sequence. This scheme is 
controlled by a so called "rate-distortion" algorithm. This is a mechanism that 
controls the rate and/or the distortion of the video coding process. In particular it 
is possible to define an approximate rate and the "Spatial Video CODEC" will 
10 adapt the quantization step so as to provide the minimum distortion for the given 
rate. Note that the distortion may vary through time according to the complexity of 
the input scene and the performances of the "Spatial Video CODEC". However, 
the rate of the compressed sequence will remain approximately constant which is 
useful to stream video content over a band-limited network. When a Uve or single 
pass encoding is necessary, the "rate-distortion" algorithm must control the "Spa- 
tial Video CODEC" through a feedback loop. The results of the previous encoded 
frames is used to predict the difficulty of encoding the following frame. Given this 
prediction, the "rate-distortion" algorithm fixes the quantization step so as to ap- 
proximate the requested rate. The rate distortion algorithm may also fix the distor- 
20 tion (or the quality) of the encoded video while the rate is free to vary along time. 
In this case, a fixed quantization associated to the requested distortion is usually 
defined and let unchanged along the sequence. Clearly, combination of the two 
objectives is also possible. 

25 The technique proposed in this document can be applied in the same context as the 
one described above. The scheme is the same as the one in Figure 2, except for the 
spatial redundancy module that is now substituted by the "Dynamic Coding" 
module as depicted in Figure 3. The input frame can be a natural image: Intra 
frame Q. frame); or the difference between two (or more) natural images: predicted 
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frame (P or B frame). The dynamic coding scheme can be applied either on the 
whole input, or independently on any rectangular sub-block of the input. The size 
of any sub-block may vary from 2x2 pixels to the size of the frame itself. The sub- 
blocks are non overlapping but they are all together covering the whole frame. 

5 

In order to optimally exploit the spatial redundancies, it is possible to evaluate not 
only one single transfonn (as the DCT in the standard approaches) associated to a 
unique quantization and entropic coder, but an arbitrary number of other "Spatial 
Video codecs" composed of any possible transform with corresponding quanti- 

10 zation and entropic coder. As in a standard approach, a rate-distortion algorithm is 
used to provide to the Dynamic Coding block an indication on the expected distor- 
tion and rate. Li our implementation, this indication comes in the form of a quality 
parameter Q that defines the quality of the encoded frames. When jQxing the qual- 
ity parameter Q, the rate will vary according to the statistical properties of the in- 

15 put signal, otherwise, it is possible to chose the right Q in order to provide a con- 
stant bitrate coded video stream. 

The basic "Dynamic Coding" block is illustrated in Figure 4. Where the input can 
be either an I frame or a P/B frame, or any rectangular sub-block of the I or P/B 

20 frame, hi Figure 4, each SCn block represents a different "Spatial Video CODEC". 
Given the input and the quality parameter Q, each SQ, block outputs a bit-stream 
and two values: flie rate R„ and the distortion D„ associated with it. So, in this 
scheme N "Spatial Video CODECs" (SC„) are evaluated. Each SQ, given its input 
and the quality parameter Q, returns a bitstream Bn and the corresponding distor- 

25 tion D„ and rate R„. According to these two values a decision is made on which 
SC„ has the best performance and it will be selected to actually encode the input. 

In this document we propose a new procedure that is able to efficiently compare 
the performances of different "Spatial Video CODECs" in two steps: Normaliza- 
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tion and Evaluation. The normalization step is perfonned offline, while the 
evaluation step is performed on the output of each SCn. hi the normalization step, 
all the "Spatial Video CODECs" that are to be evaluated are aligned in terms of 
the quality parameter Q. In the evaluation step, the rate-distortion performances of 
5 each normahzed "Spatial Video CODEC" are predicted and the one with the best 
rate-distortion ratio is selected. The exact normalization and evaluation procedure 
are detailed hereafter. 

The normalization step requires the definition of a quality parameter Q. The pa- 
10 rameter Q must have a minimum value and a maximum value. In the preferred 
implementation, Q is any integer from 0 to 100. Q is the result of a rate-distortion 
algorithm that controls the encoding process to obtain one of the following: 

1) a constant quality compression (in this case the distortion introduced by 
the coding process remains constant while the rate may vary along the 

15 sequence). 

2) a constant rate compression (in this case, the rate of the encoded se- 
quence remains constant while the quality/distortion may vary along the 
sequence). 

20 All the "Spatial Video CODECs" are normalized or "aligned" as a function of the 
same parameter Q (quality) so as to provide the same distortion for the same value 
of Q. i.e. for the same input and the same Q parameter, all normalized "Spatial 
Video CODECs" should provide a compressed frame with similar distortion, but 
possibly differing rate. 

25 

In the proposed implementation, the "Spatial Video CODECs" are aligned accord- 
ing to the MSE (Mean Square Error) distortion measure. The alignment is per- 
formed by defining the following relationship between Q and the MSE distortion 
measure: 
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12 

whereto; is a function of the quality parameter Q. As described by Mallat in 
"Analysis of Low Bit Rate Image Transform Coding" (In IEEE Transactions on 
Signal Processing, VOL. 46. No. 4, April 1998), for Spatial CODECs using a uni- 
5 form quantization, the relationship between the quality parameter Q and the quan- 
tization step A can be expressed as: /(Q) = A. In case of non-unifonn quantiza- 
tion the relationship between MSB and Q has to be respected for each Q. 

In a prefen-ed implementation, in order to perform the normalization, the "Spatial 
10 Video codecs" are uniformly quantized with a step A defined as: 

By combining equation (1) and (2) we obtain that the distortion expressed as the 
MSE is function of Q defined by: 

MSE = — = r(Q) = ^ L 

12 ^ 12 

15 where Ci controls the minimal and maximal quality and C2 the variation of the 
distortion according to Q. In particular the following values have been chosen: 
Ci = 5 and C2 = 24. This means that the distortion is doubled for each decrease of 
24 of the Q parameter. 

20 The proposed normalization procedure is not exact, but as we show in Figure 5, it 
is possible to obtain on different input signals a similar distortion for a given qual- 
ity parameter Q and this for different "Spatial Video CODECs". Four transforms 
are displayed : the Wavelet 53, Wavelet Haar, DCT and DPCM. According to a 
given Q parameter, they introduce a similar distortion. 



wo 2004/056124 



10 



PCT/IB2003/00S852 



Moreover we claim that the error En in the ahgnment of the /i-th "Spatial Video 
CODEC" is small compared to the predicted distortion: 

D„=fiQhE„,\E„\«\D„\ 



We have statistically evaluated equation (4) and the performances of the proposed 
"Spatial Video CODEC" alignment and we rqjort the results in Figure 6. Below 
the computed and predicted distortions are compared with the following relative 
measure: 

^ _ D„-MSE 
MSB 

where the real distortion measure is Dp and the predicted distortion measure is 
MSE. It turns out that most (>98% of the evaluated tests) of the predicted distor- 
tions introduce an approximation below 20% of the exact distortion. 

The evaluation step requires the computation of the rate R and of the distortion D 
for each Noraialized "Spatial Video CODEC" given a selected quality parameter 
Q and the cuirent input block to be coded. 

According to these values a decision is made on which "Spatial Video CODEC" 
has the best rate-distortion performance. This decision may be taken independ- 
ently on the rate or on the distortion values. In the first case, the "Spatial Video 
CODEC" with minimum rate will be selected in the second, the "Spatial Video 
CODEC" with the minimum distortion will be selected. However, a better deci- 
sion is obtained if both the rate and the distortion are taken into account. This is 
possible by applying a Lagrangian optimization of the two values: 
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with n [2,N] representing the index of an evaluated "Spatial Video CODEC" over 
the N total number of "Spatial Video CODECs" and X representing the Lagrangian 
multiplier that provides the relative weight between rate and distortion in selecting 
5 the best compromise. In this context, the selection of the best "Spatial Video 
CODEC" is done by choosing the one with the mininmm La. 

In order to perform a Lagrangian optimization, it is necessary to compute the rate, 
the distortion and the optimal X for each "Spatial Video CODEC" . In the follow- 
10 ing we describe the procedure adopted in this invention. 

The Lagrange multiplier is responsible for weighting the contribution of the rate 
and of the distortion in the overall evaluation of the performances of a "Spatial 
Video CODEC". Its value should be computed acconling to the rate-distortion 
15 curves of each "Spatial Video CODEC" as a function of a given input. This is 
clearly too computationally expensive to be considered in a practical application. 
In our approach we find an approximation for X that is a function of the quality 
parameter Q. The starting point is the model of high rate proposed by Mallat. This 
model states that: 

20 i) = *-2-", 

where A: is a constant depending on the "Spatial Video CODEC" and the input 
signal. From equation (7) we defines the relationship between rate and distortion 
as: 




2 



wo 2004/056124 



12 



PCT/IB2003/005852 



Merging (8) and (6) we obtain: 




We can now minimize L as a function of the only distortion D. To do this, we dif- 
ferentiate over D and we find the roots; 

dL _ 1 

dD D2\n{^f 

The final solution is: 



2\n{2)D 

If the model of distortion of equation (3) is assumed valid, equation (1 1) may be 
written as : 

ln(2) 

Equation (12) defines the optimum X as a function of the quality parameter Q. By 
referring to the scheme of Figure 4, the single parameter Q controls both the en- 
coding perfomiances of each "Spatial Video CODEC" and the Lagrange multiplier 
X. This is the strong interest and novelties of this approach: given Q, it is possible 
to measure R and D for each Normalized "Spatial Video CODEC" and at the same 
time to compute the corresponding X. The selected Normalized "Spatial Video 
CODEC" is the one that minimize L in equation (6). 

Givai the parameter Q, and the input I, computing R and D can be performed in 
an precise, but computationally expensive way by applying the "Spatial Video 
CODEC" to the input and measuring the size of the encoded stream and the intro- 
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duced distortions. This procedure is summarized in Figure 7. Note that this proce- 
dure introduces a significant amount of computations that increase with the num- 
ber N of evaluated "Spatial Video CODECs". In fact, in order to evaluate the dis- 
tortion of a single "Spatial Video CODEC" it is necessary to scale back the quan- 
5 tized coefficients, to perfomi the inverse transform and to compute the distortion 
compared to the input. The rate is obtained first by perforating the Entropic Cod- 
ing of the quantized coefBcients and then by counting the resulting bits. 

In the preferred implementation, an approximate prediction of both R and D is 
10 obtained without the need of performing the Quantization, the Entropic Coder, the 
Scaling and the Inverse Transform steps. The prediction can be computed in a 
much more computationally efficient way and the introduced approximation does 
not affect the correct choice of the best "Spatial Video CODEC". 

15 In the preferred implementation, the rate is estimated as a linear function of the 
number of zeros obtained after quantization of the coefficients while the distortion 
is approximated fi-om the distribution of the transformed coefficients. In particu- 
lar, before quantization, the histogram of the transform coefficient is computed. 
The rate is predicted as a fimction of the quantization step A: 

20 R=ayN^ 

where Nxi is the number of coefficient with an amplitude equal to Xj and the pa- 
rameter a is derived from experimental results. Note that in a preferred implemen- 
tation A is related to Q by equation (2), thus the rate is a simple fimction of the 
quality parameter Q defined by the rate-distortion algorithm. 

25 

The distortion is predicted from the distribution of the transformed coefficients: 
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where Xi is the amplitude of the coefficients and Nxi is the number of coefficient 
with an amplitude of Xj. Note that in a preferred implementation A is related to Q 
by equation (2), thus the distortion is also a simple function of the quality parame- 
5 ter Q defined by the rate-distortion algorithm. 

The proposed procedure to estimate the rate and the distortion of the "Spatial 
Video CODECS" is displayed in Figure 8. Note that in this scheme the displayed 
block does not provide the final bitstream B„, but only the transform coefficients. 

10 To obtain the final bitstream it is necessary to perform a quantization of the coef- 
ficients followed by an entropic coding. The possibility of predicting the rate and 
the distortion without performing the quantization and the entropic coding is an 
important advantage fi-om a computational point of view. In fact, these two steps 
and in particular the netropic coder are computational expensive. In our imple- 

15 mentation all the "Spatial Video CODECs" are evaluated, but only the one se- 
lected requires the execution of the quantization and of the entropic coding. Com- 
pared to Figure 7, now only the transform is necessary to estimate the distortion 
and the rate of a Spatial transform. Note that when the best "Spatial Video 
CODEC" is selected, it will be necessary to quantize the transform coefficients 

20 and to apply the coiresponding entropic coder. 

Li order to illustrate the principle of the dynamic coder, two examples of measure 
of the performance will be presented. Both example have been computed using a 
frame based dynamic coder. The first example shown in Figure 9 presents the in- 
25 dividual performance of three types of "Spatial Video CODEC". The first one is a 
wavelet based CODEC, well adapted for natural scene, the second one is a wave- 
let based CODEC well adapted to discontinuities, and the last one is a block based 
CODEC well ad^ted to code local information. Depending of the characteristic 
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of the scene one "Spatial Video CODEC" outperforms the others. However, it can 
be seen that the optimal "Spatial Video CODEC" between frames 450 and 500 
becomes sub optimal in the following frames. The dynamic coder will always 
choose the best "Spatial Video CODEC" in terms of MSE, providing an optimum 
5 overall performances. 

The next example (Figure 10) shows in which proportion each "Spatial Video 
CODEC" scheme was chosen in order to compress a typical TV sequence of ap- 
proximately 10 minutes at 1000 kbits per second. Again we confirai that there is 
10 not a single "Spatial Video CODEC" that significantly out-performs all other 
"Spatial Video CODECs." 

The proposed dynamic coder offers several advantages over standard approaches: 

1) It guarantees the choice of an optimal encoding with high granularity: each 
block 2x2 can be encoded with a different "Spatial Video CODEC". 

2) Its complexity is hmited thanks to the prediction of the rate and the distor- 
tion from the transform coefficients. 

3) It is an open architecture: it extend the standard approaches to an unlimited 
niunber of "Spatial Video CODECs". 

4) It offers a high degree of adaptation: the best "Spatial Video CODEC" is 
automatically selected according to the input and the target rate. 

5) According to the application the complexity may be reduced simply by re- 
ducing the number of "Spatial Video CODECs" to be evaluated. This re- 
duction or increase could be performed at run-time. 

6) It allows to introduce application specific "Spatial Video CODECs" with- 
out loosing the generality of standard approaches. 



20 



The dynamic codec is particularly suitable for all those application where high 
quality video signals have to be encoded in real-time at low-bit rates (< 500 Kbps) 
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for interactive TV, Corporate TV, TV over ADSL, TV broadcast. In particular for 
all those input signals characterized by the presence of natural, synthetic and 
mixed scenes. 



